An augmented reality visor for surgical navigation was developed and clinically assessed in maxillofacial surgery within the framework of the H2020 project coordinated by UNIPI (VOSTARS), which has been optimize in the POR-CREO ARTS4.0 project coordinated by Orthokey. In this activity, the current prototype will be engineered both on on the SW side (by UNIPI) and on the HW side (by Orthokey). The final platform will be then clinically tested in regional university hospitals in three two different surgical specialties: orthopaedic surgery and neurosurgery. Within this activity, the UNIPI DII researchers aim to implement a software platform capable of deploying AR-based applications for surgical navigation on different visualization interfaces and relying on different sensing devices. The software framework will have to be highly modular and efficient in terms of end-to-end latency. The visor, as in the previous versions will offer both optical and video see-through vision modalities.
SW development by UNIPI
As UNIPI group, we performed an in-depth analysis of the most efficient and flexible tools in terms of computer-vision and computer graphics libraries to be adopted considering that the augmented reality software framework must be compliant with the project requirements in terms of its compatibility with different visualization and sensing devices, modularity, ease-of-development, and performance constraints. To this end, we first analysed the most suitable state-of-the-art software development tools and libraries for computer vision, computer graphics, and augmented reality, and we made the following outline choices:
- Windows OS (for its ease-of-development and widespread adoption).
- Microsoft Visual Studio IDE (for its ease-of-development and effectiveness).
- Unity3D game engine (for its ease-of-development and modularity).
- OpenCV machine Vision Library (for its compatibility, ease-of-development, performance constraints).
- NVIDIA CUDA library for harnessing the multicore capability of the dedicated graphic card (for its ability to leverage the multi-core architecture of the dedicated NVIDIA GPU).
Specifically, the architecture of the AR platform can be summarized as follows:
The main AR app will be implemented under UNITY3D game engine environment in c#. In the app, the main AR rendering cycle will run. The underlying libraries for handling the computer vision applications will be integrated in the app as three separated dynamic libraries (DLL) built in c++ (i.e., plugin libraries in the UNITY environment). The three DLLs can potentially be integrated into non-Unity AR applications that require the same acquisition and localization features. The DLLs are implemented in a way to be effective and efficient in AR applications that is low latency between image acquisition and rendering of virtual content.
HW development by Orthokey
HMD is undergoing an electronical and mechanical engineering process:
- Performed EM tests based on EN60601-1-2: the conducted and radiated emissions are higher than the values indicated in the standard.
- It is necessary to contact an expert consultant in microelectronics to evaluate if the HMD optic, especially the controller, can be shielded or need a total re-engineering
Spine intervention:
- The HMD will be paired with Orthokey medical device.
- A specific frame has been identified to simultaneously detect tracking markers by HMD and Orthokey navigation System.
- A specific workflow will be defined for the landmark acquisition on a vertebra and a registration algorithm will be implemented
- Surgeons’ involvement